llama: disable fused up-gate when an offload backend lacks GGML_OP_FUSED_UP_GATE (fixes silent 3.5x Metal slowdown)#2133
Conversation
GGML_OP_FUSED_UP_GATE has a CUDA implementation but no Metal one. With full offload on Apple Silicon the scheduler silently assigns every fused up-gate node to the CPU backend, forcing a GPU<->CPU round trip on every FFN layer (24x per token on Qwen 2.5-0.5B) with no error or warning. Measured cost on M2: 36 tok/s vs 125 tok/s decode (3.5x). Probe the offload backends at context creation with a dummy fused up-gate node; if any backend does not support the op, disable cparams.fused_up_gate with a warning so the graph builds the decomposed MUL_MAT + FUSED_MUL_UNARY form and stays on GPU. Backend-agnostic: any backend lacking the kernel gets the same fallback, and the probe stops triggering once the op gains a native kernel.
|
We verified it on M2. We built this branch with Metal (Apple M2, macOS) and ran Qwen 2.5-0.5B-Instruct Q4_K_M with full offload (
Two pre-existing Metal/ARM build issues we hit while compiling this branch on Xcode clang 21, both unrelated to this PR:
|
|
The Metal back-end has not been maintained for a while. There are several other new ops that are missing in addition to the fused up/gate op. Are you interested in bringing the Metal back-end up to speed and helping maintain it? The way out of the missing |
|
We can't commit to formally maintaining the Metal back-end right now, but we can do something in between: we run ik_llama.cpp on Apple Silicon daily, so we're happy to upstream the missing ops as we hit them. First concrete offer: we've already implemented the missing ROPE_MULTI (mrope/imrope) kernels for Metal in our fork (F32/F16 pipelines, section-based position handling, vision mode asserted out), tested on an M2. If you're interested we'll clean that up into a PR against main. Same goes for other ops we find missing during our own testing — we'd send those as individual PRs rather than a big drop. One scope caveat: everything we build and test covers M1–M4 (unified memory, current Metal feature set). The M5 generation reportedly changes things (new GPU/neural-accelerator paths), and we don't have an M5 machine — so anything we contribute should be read as validated on M1–M4 only. Meanwhile we'll test #2137 on our M2 and report back on #2135. |
|
The mrope/imrope Metal kernels mentioned above are now up as #2140 — didn't want to gate that on this thread. Validated on M2 (kernel output matches CPU; Qwen 3.5-9B full-offload PPL within 0.006 of the CPU baseline). |
2 participants
Problem
GGML_OP_FUSED_UP_GATEis implemented in the CUDA backend (ggml_cuda_up_gate_unary) but not in the Metal backend. With-ngl 99on Apple Silicon, the backend scheduler silently assigns every fused up-gate node to the CPU backend, forcing a GPU→CPU→GPU synchronization stall on every FFN layer — 24 times per token on Qwen 2.5-0.5B.There is no error and no warning; inference is just silently slower. Measured on an M2 (Qwen 2.5-0.5B, Q5_0, full offload):
fused_up_gate = true)--no-fused-up-gateThis affects all standard transformer models (Qwen 2.5, Llama, DeepSeek, ...) on all Apple Silicon Macs (M1–M4). Hybrid DeltaNet models are unaffected (their FFN path doesn't emit this op).
Fix
At context creation, after the backends are initialized, build a dummy
GGML_OP_FUSED_UP_GATEnode and probe each non-CPU backend withggml_backend_supports_op(). If any offload backend lacks the op, disablecparams.fused_up_gateand log a warning. The graph then builds the decomposed form (MUL_MAT+FUSED_MUL_UNARY), which Metal handles natively, and the whole graph stays on GPU.This deliberately does not special-case Metal: any backend without the kernel (Vulkan, SYCL, ...) gets the same fallback, and once someone adds a Metal kernel for the op the probe simply stops triggering.
--no-fused-up-gatestill works as an explicit override; the probe only ever turns the option off, never on.Testing
fused_up_gate = 1in the context log and normal generation).fused_up_gate = falseunder Metal full offload) is the same mitigation we have been running manually via an eval callback on M2 since April — that is where the 36→125 tok/s numbers come from.I don't have Apple hardware in CI reach for this branch, so the probe path on Metal is verified by inspection.Update: we verified it on M2 (details in comment below): this branch built withGGML_METAL=ON, Qwen 2.5-0.5B Q4_K_M at-ngl 99— the probe warning fires at context creation and default decode runs at 112.5 t/s, matching the explicit--no-fused-up-gaterun (115.3 t/s, identical output) instead of the previous ~36 t/s CPU-fallback behavior.